import s3fs
import xarray as xr
import hvplot.xarray
import geoviews as gv
import datashader as dsh
from holoviews.operation.datashader import rasterize
gv.output(size=300)Visualize Zarr
xarray and dask with data from SMAP
Run this notebook
You can launch this notebook in VEDA JupyterHub by clicking the link below.
Launch in VEDA JupyterHub (requires access)
Learn more
Inside the Hub
This notebook was written on the VEDA JupyterHub and as such is designed to be run on a jupyterhub which is associated with an AWS IAM role which has been granted permissions to the VEDA data store via its bucket policy. The instance used provided 16GB of RAM.
See (VEDA Analytics JupyterHub Access)[https://nasa-impact.github.io/veda-docs/veda-jh-access.html] for information about how to gain access.
Outside the Hub
The data is in a protected bucket. Please request access by emailng aimee@developmentseed.org or alexandra@developmentseed.org and providing your affiliation, interest in or expected use of the dataset and an AWS IAM role or user Amazon Resource Name (ARN). The team will help you configure the cognito client.
You should then run:
%run -i 'cognito_login.py'
Approach
This notebook demonstrates 2 strategies for to subselect data from a Zarr dataset in order to visualize using the memory of a notebook.
- Downsample the temporal resolution of the data using
xarray.DataArray.resample - Coarsening the spatial aspect of the data using
xarray.DataArray.coarsen
A strategy for visualizing any large amount of data is Datashader which bins data into a fixed 2-D array. The call to rasterize ensures the use of the datashader library to bin the data.
About the data
The SMAP mission is an orbiting observatory that measures the amount of water in the surface soil everywhere on Earth.
Load libraries
Optional: Create and Scale a Dask Cluster
We create a separate Dask cluster to speed up reprojecting the data (and other potential computations which could be required and are parallelizable).
Note if you skip this cell you will still be using Dask, you’ll just be using the machine where you are running this notebook.
from dask_gateway import GatewayCluster, Gateway
gateway = Gateway()
clusters = gateway.list_clusters()
# connect to an existing cluster - this is useful when the kernel shutdown in the middle of an interactive session
if clusters:
cluster = gateway.connect(clusters[0].name)
else:
cluster = GatewayCluster(shutdown_on_close=True)
cluster.scale(16)
client = cluster.get_client()
clientOpen the dataset from S3
s3 = s3fs.S3FileSystem()
root = 'veda-data-store-staging/EIS/zarr/SPL3SMP.zarr'
store = s3fs.S3Map(root=root, s3=s3)
ds = xr.open_zarr(store=store)
ds<xarray.Dataset>
Dimensions: (northing_m: 406, easting_m: 964,
datetime: 1679)
Coordinates:
* datetime (datetime) datetime64[ns] 2018-01-01 ... 2...
* easting_m (easting_m) float64 -1.735e+07 ... 1.735e+07
* northing_m (northing_m) float64 7.297e+06 ... -7.297e+06
Data variables: (12/26)
albedo (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
albedo_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
bulk_density (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
bulk_density_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
clay_fraction (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
clay_fraction_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
... ...
static_water_body_fraction (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
static_water_body_fraction_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_flag (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_flag_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_temperature (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_temperature_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>Select the variable of interest (soil moisture for this example).
soil_moisture = ds.soil_moisture
soil_moisture<xarray.DataArray 'soil_moisture' (northing_m: 406, easting_m: 964,
datetime: 1679)>
dask.array<open_dataset-e9352a6cb9ac62da20f04f798593159fsoil_moisture, shape=(406, 964, 1679), dtype=float32, chunksize=(100, 100, 100), chunktype=numpy.ndarray>
Coordinates:
* datetime (datetime) datetime64[ns] 2018-01-01 2018-01-02 ... 2022-09-09
* easting_m (easting_m) float64 -1.735e+07 -1.731e+07 ... 1.735e+07
* northing_m (northing_m) float64 7.297e+06 7.26e+06 ... -7.26e+06 -7.297e+06
Attributes:
coordinates: /Soil_Moisture_Retrieval_Data_AM/latitude /Soil_Moisture_Re...
long_name: Representative DCA soil moisture measurement for the Earth ...
units: cm**3/cm**3
valid_max: 0.5
valid_min: 0.019999999552965164Strategy 1: Downsample the temporal resolution of the data
To plot one day from every month, resample the data to 1 observation a month.
somo_one_month = soil_moisture.resample(datetime="1M").nearest()Quick plot
We can generate a quick plot using hvplot and datashader.
# workaround to avoid warnings that are triggered within Dask.
import warnings
warnings.filterwarnings("ignore", message="All-NaN slice encountered", category=RuntimeWarning)somo_one_month.hvplot(
x="easting_m",
y="northing_m",
groupby="datetime",
crs="epsg:6933",
coastline=True,
rasterize=True,
aggregator="mean",
frame_height=150,
widget_location="bottom"
)WARNING:param.main: Calling the .opts method with options broken down by options group (i.e. separate plot, style and norm groups) is deprecated. Use the .options method converting to the simplified format instead or use hv.opts.apply_groups for backward compatibility.
Reproject before plotting
Reproject the data for map visualization.
somo_one_month = somo_one_month.transpose('datetime', 'northing_m', 'easting_m')
somo_one_month = somo_one_month.rio.set_spatial_dims(x_dim="easting_m", y_dim="northing_m")
somo_one_month = somo_one_month.rio.write_crs("epsg:6933")
somo_reprojected = somo_one_month.rio.reproject("EPSG:4326")
somo_reprojected<xarray.DataArray 'soil_moisture' (datetime: 57, y: 1046, x: 2214)>
array([[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
...
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)
Coordinates:
* x (x) float64 -179.9 -179.8 -179.6 -179.4 ... 179.6 179.8 179.9
* y (y) float64 84.96 84.8 84.64 84.48 ... -84.64 -84.8 -84.96
* datetime (datetime) datetime64[ns] 2018-01-31 2018-02-28 ... 2022-09-30
spatial_ref int64 0
Attributes:
coordinates: /Soil_Moisture_Retrieval_Data_AM/latitude /Soil_Moisture_Re...
long_name: Representative DCA soil moisture measurement for the Earth ...
units: cm**3/cm**3
valid_max: 0.5
valid_min: 0.019999999552965164Note that this is now a fully materialized data array - when we reproject we trigger an implicit compute.
Create a geoviews dataset and visualize the data on a map.
kdims = ['datetime', 'x', 'y']
vdims = ['soil_moisture']
xr_dataset = gv.Dataset(somo_reprojected, kdims=kdims, vdims=vdims)
images = xr_dataset.to(gv.Image, ['x', 'y'])
rasterize(images, precompute=True, aggregator=dsh.mean('soil_moisture')) * gv.feature.coastlineStrategy 2: Coarsen spatial resolution of the data
Below, we coarsen the spatial resolution of the data by a factor of 4 in the x and 2 in the y. These values were chosen because they can be used with the exact boundary argument as the dimensions size is a multiple of these values.
You can also coarsen by datetime, using the same strategy as below but replacing easting_m and northing_m with datetime. If {datetime: n} is the value give to the dim argument, this would create a mean of the soil moisture average for n days.
Once the data has been coarsned, again it is reprojected for map visualization and then visualized using Geoviews.
coarsened = soil_moisture.coarsen(dim={"easting_m": 4, "northing_m": 2}).mean()
coarsened = coarsened.transpose('datetime', 'northing_m', 'easting_m')
coarsened = coarsened.rio.set_spatial_dims(x_dim="easting_m", y_dim="northing_m")
coarsened = coarsened.rio.write_crs("epsg:6933")
coarsened_reprojected = coarsened.rio.reproject("EPSG:4326")
coarsened_reprojected<xarray.DataArray 'soil_moisture' (datetime: 1679, y: 315, x: 667)>
array([[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
...
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)
Coordinates:
* x (x) float64 -179.7 -179.2 -178.7 -178.1 ... 178.6 179.2 179.7
* y (y) float64 84.77 84.23 83.7 83.16 ... -83.64 -84.18 -84.72
* datetime (datetime) datetime64[ns] 2018-01-01 2018-01-02 ... 2022-09-09
spatial_ref int64 0
Attributes:
coordinates: /Soil_Moisture_Retrieval_Data_AM/latitude /Soil_Moisture_Re...
long_name: Representative DCA soil moisture measurement for the Earth ...
units: cm**3/cm**3
valid_max: 0.5
valid_min: 0.019999999552965164
_FillValue: 3.402823466e+38kdims = ['datetime', 'x', 'y']
vdims = ['soil_moisture']
xr_dataset = gv.Dataset(coarsened_reprojected, kdims=kdims, vdims=vdims)
images = xr_dataset.to(gv.Image, ['x', 'y'])rasterize(images, precompute=True, aggregator=dsh.mean('soil_moisture')) * gv.feature.coastlineCleanup
When using a remote Dask cluster it is recommented to explicitly close the cluster.
client.shutdown()